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1 Introduction 

Connectomics is a field of neuroscience that analyzes neuronal connections. A con- 
nectome is a complete map of a neuronal system, comprising all neuronal connec- 
tions between its structures. The term 'connectome' is close to the word 'genome' 
and implies completeness of all neuronal connections, in the same way as a genome 
is a complete listing of all nucleotide sequences. The goal of connectomics is to 
create a complete representation of the brain's wiring. Such a representation is be- 
lieved to increase our understanding of how functional brain states emerge from 
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their underlying anatomical structure |89|. Furthermore, it can provide important 
information for the cure of neuronal dysfunctions like schizophrenia or autism [84 1. 

Different types of connectivity can be distinguished. Structural or anatomical 
connectivity usually refers to the "wiring diagram" of physical connections between 
neural elements. These anatomical connections range in scale from those of local 
circuits of single cells to large-scale networks of interregional pathways (881 . Func- 
tional connectivity is defined as "the temporal correlation between spatially remote 
neurophysiological events" (33). This can be seen as a statistical property; it does 
not necessarily imply direct anatomical connections. Finally, effective connectivity 
concerns causal interactions between distinct units within a nervous system (33). 

Sporns et al. (89;| differentiate between macro-, meso- and microscale connec- 
tomes. At the macroscale a whole brain can be imaged and divided into anatomically 
distinct areas that maintain specific patterns of interconnectivity. Spatial resolution 
at the macroscale is typically in the range of millimeters. One order of magnitude 
smaller is the mesoscale connectome that describes connectivity in the range of mi- 
crometers. At this scale, local neuronal circuits, e.g., cortical columns, can be distin- 
guished. At the finest microscale the connectome involves mapping single neuronal 
cells and their connectivity patterns. Ultimately connectomes from all scales should 
be merged into one hierarchical representation (89) . 

Independently of the scale, the connectivity can be represented as a brain graph 
G(N;E) with nodes N and weigthed edges E representing anatomical entities and 
the degree of structural or functional interactions, respectively. Associated to each 
abstract graph is a graph in real space that connects real anatomical entities. Neu- 
ral systems can be investigated by analyzing topological and geometrical proper- 
ties of these graphs and by comparing them. An equivalent way of representing an 
undirected or directed brain graph is a connectivity or association matrix C, whose 
entries c,y represent the degrees of interactions. Thresholding and sometimes also 
binarizing them reveals the essential interactions. A spatial connectivity graph can 
be depicted in real space, showing the actual physical structure of the neural system. 
A connection matrix is usually visualized using a color-encoding matrix view. For 
more details and examples see, e.g., the recent reviews (TOpl) . 

In contrast to genomics, the field of connectomics is to a large extent based on 
image data. Therefore visualization of image data can directly support the analysis 
of brain structures and their structural or functional connections. 

In this Chapter, we review the current state-of-the-art of visualization and im- 
age processing techniques in the field of connectomics and describe some remain- 
ing challenges. After presenting some biological background in Section [2] and an 
overview of relevant imaging modalities in Section [3] we review current tech- 
niques to extract connectivity information from image data at macro-, meso- and 
microscale in Sections 4jj6 Section [7] focuses on integration of anatomical connec- 
tivity data. The last section discusses visually supported analysis of brain networks. 
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2 Biological Background 



Neural systems. Functionally, neurons (or nerve cells) are the elementary signal- 
ing units of the nervous system, including the brain. Each neuron is composed of 
a cell body (soma), multiple dendritic branches and one axonal tree, which receive 
input from and transfer output towards other neurons, respectively. This transfer is 
either chemical (synapses) or electrical (gap junctions). Generally, during synaptic 
transmission vesicles containing neurotransmitter molecules are released from ter- 
minals (boutons) on the axon of the presynaptic neuron, diffuse across the synaptic 
cleft, and are bound by receptors on dendritic spines of the postsynaptic neuron, 
inducing a voltage change, i.e., a signal. 

These basic building blocks can mediate complex behavior, as potentially large 
numbers of them are interconnected to form local and long-range neural microcir- 
cuits. At the meso-level, local neuron populations, e.g., cortical minicolumns, can 
be identified that act as elementary processing units. At the macroscale, neurons in 
the human cortex are arranged in a number of anatomically distinct areas, connected 



by interregional pathways called tracts [89 1. 



Model systems. An important neuroscientific goal is to understand how the human 
brain works. However, due to its complexity (with an estimated 10 11 neurons with 



10 15 connections B89J), brain function at the circuit or cellular level is often studied 
in other organisms that are more amenable in complexity and size. 

Conserved genes and pathways between different species offer the potential 
of elucidating the mechanisms that affect complex human traits based on similar 
processes in other organisms. This problem is particularly tractable in the round- 
worm Caenorhabditis elegans, whose brain with 302 neurons has been completely 
mapped |104|, or in insects. In these organisms brain structure and function can 
be studied at the level of single identifiable neurons. Classical insect model or- 
ganisms that are well understood and allow easy genetic manipulations are fruit 
fly Drosophila melanogaster and the honeybee. Drosophila, for example, has been 
shown to be an experimentally amenable model system even for the study of such 
quintessential human physiological traits as alcoholism, drug abuse, or sleep ]64) . 

Rodents, being mammals, have a brain structure that is similar but much smaller 
than the human brain, and that therefore can be used to study cortical networks. 
The mouse brain is an attractive model system to study, for example, the visual 
system, due to the abundant availability of genetic tools allowing monitoring and 
manipulating certain cell types or circuits [39) . The whisker-barrel pathway of the 
rat is a relatively small and segregated circuit that is amenable to studying sensory 
information processing at the molecular/synaptic, cell, and circuit/region level. 
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3 Imaging Modalities Employed in Connectomics 



We now provide an overview of imaging modalities that are used in obtaining con- 
nectivity information. They differ in the spatial and temporal resolution at which 
connectivity is captured. At the macroscale there is a wide range of structural and 
functional imaging modalities, with applications in medical settings and anatomi- 
cal research. Functional imaging modalities include electroencephalography (EEG), 
magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), 
and positron emission tomography (PET). Modalities like single-photon emission 
computed tomography (SPECT) and magnetic resonance imaging (MRI) provide 
structural information on the macroscale. Section [4] gives a detailed introduction to 
the relevant modalities in the context of connectomics. At the mesoscale light mi- 
croscopy (LM) techniques provide sufficient resolution to image single neurons. 
Most light microscopy techniques focus on structural imaging. Techniques like 
wide-field fluorescence microscopy allow for imaging of living cells, and compu- 
tational optical sectioning microscopy techniques [18| enable non-destructive ac- 
quisition of 3D data sets. Section |5]provides further details about light microscopy 
techniques. At the microscale the sufficient resolution is offered by electron mi- 
croscopy techniques (EM) such as Transmission Electron Microscopy (TEM) and 
Scanning Electron Microscopy (SEM). These methods require technically complex 
specimen preparation and are not applicable to live cell imaging. Imaging of 3D 
volumes requires ultra-thin sectioning of the brain tissue followed by computational 
realignment of the acquired images into one image volume [47]. More information 
about electron microscopy in the connectomics setting can be found in Section [6] 
Figure [T] provides an overview of the different imaging modalities and their spatial 
and temporal resolution. 
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Fig. 1 Different brain imaging modalities and their spatial and temporal resolutions. For con- 
nectomics, light- (LM) and electron microscopy (EM) are mostly performed in vitro. The color 
indicates functional vs. structural information in the acquired data. 
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First, we discuss the main acquisition techniques for revealing macroscopic func- 
tional and structural connectivity. We start with MEG and EEG, as these were used 
for functional connectivity before fMRI, then diffusion-weighted MRI for structural 
connectivity, and finally fMRI for functional connectivity. Besides the visualization 
approaches discussed here, the reader is also referred to Section [8] for more detail 
on network analysis and comparative visualization techniques. 



4.1 EEG and MEG 

Developed in the 1920s, electroencephalography (EEG) is the oldest noninvasive 
functional neuroimaging technique, which records electrical brain activity from 
electrodes on the scalp. Nowadays, the number of electrodes can be as large as 



128 or even 512; in that case one speaks of multichannel or high-density EEG [82 1. 
By contrast, magnetoencephalography (MEG) measures magnetic fields outside the 
head induced by electrical brain activity [36 1. The temporal frequency of these sig- 
nals ranges from less than 1 Hz to over 100 Hz. The spatial resolution is lower than 
for fMRI. Sometimes, MEG is preferred over EEG because the electrical signals 
measured by EEG depend on the conduction through different tissues (e.g., skull 
and skin). However, EEG has much lower costs and higher equipment transporta- 
bility than MEG (and fMRI). Moreover, EEG allows participants more freedom to 
move than MEG and fMRI. In Section[8]we will discuss the use of EEG to discover 
functional brain networks. Therefore, we will focus on EEG for the remainder of 
this subsection. 

Electrical potentials generated within the brain can be measured with electrodes 
at the scalp during an EEG recording. The measured EEG signals reflect rhythmical 
activity varying with brain state. Specific brain responses can be elicited by the pre- 
sentation of external stimuli. For EEG analysis, one often studies activity in various 
frequency bands, such as alpha, beta, theta or delta bands. As a result of volume con- 
duction, an electrical current flows from the generator in the brain through different 
tissues (e.g., brain, skull, skin) to a recording electrode on the scalp. The measured 
EEG is mainly generated by neuronal (inhibitory and excitatory) postsynaptic po- 
tentials and burst firing in the cerebral cortex. Measured potentials depend on the 
source intensity, its distance from the electrodes, and on the conductive properties 
of the tissues between the source and the recording electrode. 

Several visualization methods are applied to assist the interpretation of the EEG. 
In a conventional EEG visualization, the time-varying EEG is represented by one 
time series per electrode, displaying the measured potential as a function of time. 
Synchronous activity between brain regions is associated with a functional relation- 
ship between those regions. EEG coherence, calculated between pairs of electrode 
signals as a function of frequency, is a measure for this synchrony. A common vi- 
sualization of EEG coherence is a graph layout. In the case of EEG, graph vertices 
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(drawn as dots) represent electrodes and graph edges (drawn as lines between dots) 
represent similarities between pairs of electrode signals. Traditional visual repre- 
sentations are, however, not tailored for multichannel EEG, leading to cluttered rep- 
resentations. Solutions to this problem are discussed in Section|8] 



4.2 MRI 

In magnetic resonance imaging, or MRI, unpaired protons, mostly in hydrogen 
atoms, precess at a frequency related to the strength of the magnetic field applied by 
the scanner. When a radio-frequency pulse with that specific frequency is applied, 
the protons resonate, temporarily changing their precession angle. They eventually 
regain their default precession angle, an occurrence that is measured by the scan- 
ner as an electromagnetic signal. By applying magnetic field gradients throughout 
three-dimensional space, protons at different positions will precess and hence res- 
onate at different frequencies, enabling MRI to generate volume data describing the 
subject being scanned. 

4.2.1 Diffusion-Weighted Imaging 

Water molecules at any temperature above absolute zero undergo Brownian motion 
or molecular diffusion ]24| . In free water, this motion is completely random, and wa- 
ter molecules move with equal probability in all directions. In the presence of con- 
straining structures such as the axons connecting neurons together, water molecules 
move more often in the same direction than they do across these structures. When 
such a molecule moves, the two precessing protons its hydrogen nucleus contains 
move as well. When this motion occurs in the same direction as the diffusion gradi- 
ent q (an extra magnetic field gradient that is applied during scanning) of a diffusion- 
weighted MRI scan, the detected signal from that position is weakened. By apply- 
ing diffusion gradients in a number of different directions, a dataset can be built up 
showing the 3D water diffusion at all points in the volume, which in turn is related 
to the directed structures running through those points. 

Diffusion tensor imaging. When at least six directions are acquired, a 3 x 3 sym- 
metric diffusion tensor can be derived, in which case the modality is described as 
Diffusion Tensor Imaging (DTI). Per voxel DTI, often visualized with an ellipsoid, 
is not able to represent more than one major diffusion direction through a voxel. 
If two or more neural fibers were to cross, normal single tensor DTI would show 
either planar or more spherical diffusion at that point. The left image of Figure [2] 
shows a 3-D subset of such a dataset, where each tensor has been represented with 
a superquadric glyph. 

DTI visualization techniques can be grouped into the following three classes [ 100 1: 
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Scalar metrics reduce the multi-valued tensor data to one or more scalar values 
such as fractional anisotropy (FA), a measure of anisotropy based on the eigenval- 
ues of the tensor, and then display the reduced data using traditional techniques, 
for example multi-planar reformation (slicing) or volume rendering. An often-used 
technique is to map the FA to intensity and the direction of the principal tensor 
eigenvector to color and then display these on a slice. Multiple anisotropy indices 
can also be used to define a transfer function for volume rendering, which is then 
able to represent the anisotropy and shape of the diffusion tensors §5l\ . 

Glyphs can be used to represent diffusion tensors without reducing the dimen- 
sionality of the tensor. In its simplest form, the eigensystem of the tensor is mapped 
directly to an ellipsoid. More information can be visually represented by mapping 
diffusion tensors to superquadrics [50|, see also Figure [2] 

Vector- and tensor-field visualization techniques visualize global information of 
the field. The best known is probably fiber tractography, where lines are recon- 
structed that follow the tensor data in some way and hence are related to the major 
directions of neural fibers. In its simplest form, streamlines, tangent to the principal 
eigenvectors of the diffusion tensors, are extracted and displayed (2), where care 
has to be taken to terminate the streamlines in areas of isotropic or planar diffusion. 
Hyperstreamlines take into account more of the tensor information |107| . Many 
tractography approaches require one or more regions of interest to be selected be- 
fore tracts can be seeded starting only from those regions, while more recent efforts 
allow for full-brain fiber tracking followed by more intuitive interactive selection 
within the brain's tracked fiber bundles [ 8 , 86 1, see the right image in Figure[2]for an 
example. For a simplified visual representation, the envelopes of clustered stream- 
line bundles can be shown J26) , or illustrative techniques such as depth-dependent 
halos can be used (27). With probabilistic tractography, local probability density 
functions of diffusion or connectivity are estimated and can in turn be used to es- 
timate the global connectivity, that is the probability that two points in the brain 
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are structurally connected [4|. This type of data is arguably a higher fidelity rep- 
resentation of structural connectivity. Connectivity between two points can be vi- 
sualized with, e.g., constant-probability isosurfaces, with direct volume rendering 



of the probability field, or using topological methods from flow visualization [83 1. 
Calculating and effectively visualizing a full-brain probabilistic tractography would 
be challenging. 

DSI and HARDI. As explained above, DTI is not able to capture more than one 
principal direction per sample point. In order to reconstruct the full diffusion proba- 
bility density function (PDF), that is, the function describing the probability of water 
diffusion from each voxel to all possible displacements in the volume, about 500 or 
more diffusion-weighted MRI volumes have to be acquired successively. This is 
called diffusion spectrum imaging or DSI [ [35] and is the canonical way of acquir- 
ing the complete 3-D water diffusion behavior. However, the time and processing 
required to perform full DSI complicate its use in research and practice. 

In High Angular Resolution Diffusion Imaging, or HARDI, usually 40 or more 
directions are acquired in order to sample the 3-D diffusion profile around every 
point 1 94- 1 . Based on such data, multiple diffusion tensors can be fit to the data [94|, 



higher order tensors can be used 1 70 1, or a model-free method such as Q-Ball imag- 
ing (93) can be applied. Q-Ball yields as output an orientation distribution function, 
or ODF. The ODF is related to the diffusion PDF in that it describes for each direc- 
tion the sum of the PDF values in that direction. It can be visualized as a deformed 
sphere whose radii represents the amount of diffusion in the respective direction. 

HARDI visualization follows much the same lines as DTI visualization, except 
that the data are more complex. Analogous to DTI, HARDI scalar metrics, such as 
generalized (fractional) anisotropy and fractional multifiber index, can be used to 
reduce the data to one or more scalar values that can be visualized with traditional 
techniques. Multiple diffusion tensors can be represented as glyphs, or the diffusion 
ODF can be directly represented using a tesselated icosahedron or by raycasting 
the spherical harmonics describing the ODF [71]. This results in a field of complex 
glyphs representing at each point the diffusion profile at that position. In contrast to 
DTI glyph techniques, regions of crossing fibers can in general be identified. 

Although there are fewer examples, especially in the visualization literature, 
(probabilistic) fiber tracking can be performed based on HARDI data (73). More 
recently, HARDI glyphs have been combined dynamically with DTI glyphs and 
fiber tracts based on local data characteristics |76|. 



4.3 Functional MRI 



Blood-oxygen-level dependence, or BOLD, is a special type of MRI that is able to 
measure increased levels of blood oxygenation (68) . Due to requiring more glucose 
from the bloodstream, active neurons cause higher blood oxygenation in nearby 
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veins. Based on this principle, functional MRI, or fMRI, uses BOLD to image time- 



dependent 3-D neural activity in the brain 1 69 1. 

fMRI can also be used to derive functional or effective connectivity in the brain. 
Functional connectivity is determined by calculating the temporal correlations be- 
tween the fMRI signals originating from different parts of the brain [33) . This is 
done either whilst the subject performs a specific task, in order to assess how the 
brain network is applied during that task, or during resting state, in order to derive 
the baseline functional brain network. Connectivity data can be determined between 
a specific seed region or voxel and one or more other regions or voxels, or exhaus- 
tively between all regions or voxels in the brain. 

Effective connectivity, defined as the causal influence one neuronal system exerts 
over another, is dependent on a model of the connectivity between the participating 
regions. For example, the signal at one position could be expressed as the weighted 
sum of the signals elsewhere [33 1. If the model is invalid, the effective connectivity 
derived from fMRI is also invalid. 

Visualization of fMRI-derived connectivity information is quite varied, often 
combining techniques from scientific and information visualization. Scatter plots 
have been used to plot correlation strength over distance, dendrograms and multi- 
dimensional scaling to represent correlations between regions 1 8 1 1 (see Figure [3] 
left), matrix bitmaps to represent region-wise correlation matrices ]28|, 2-D and 3- 
D (pseudo-)anatomical node-link diagrams to show the derived brain networks 1 105 1 
(see Figure [3]right), and coupled-view visual analysis techniques to explore resting 
state fMRI data |97|. When connectivity is determined between all pairs of voxels 
in the cortex, visualization and knowledge extraction pose perceptual and computa- 
tional challenges that have not yet been fully explored. 




Fig. 3 On the left, different brain regions have been mapped to 2-D using multi-dimensional scal- 
ing, with some of the strongest fMRI-derived connections between them shown as lines |81 1. On 
the right, a number of the strongest fMRI-derived connections between regions are shown in their 
3-D anatomical context 1 105 1. 
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5 Mesoscale Connectivity 



Light microscopy was the first modality that allowed for imaging of single neuronal 
cells. While the resolution of a light microscope is not sufficient to resolve synapses, 
it allows identifying major cell parts, like dendrites, somas, axons, and also boutons 
as possible locations for synaptic connections. Imaging whole neuronal cells and 
analyzing their geometry enables neuroanatomists to identify different types of cells 
and to come to conclusions about their function. Following the motto "the gain in the 
brain lies mainly in the stain" [1], the three following main techniques are employed 
to map neuronal circuits with light microscopy | |6T) . 

Single-cell staining by dye impregnation. This is the oldest staining method and 
it laid the foundation for modern neuroscience. As neuronal tissue is densely packed 
with cells, a complete staining of the whole sample would not allow to discriminate 
single cells in light microscopy images. Instead, the so-called Golgi stain enables 
stochastic marking of just a few individual nerve cells. The stained cells appear 
dark in the light microscopy images, discriminating them from a bright background 
formed by the unstained tissue. This staining method combined with the ability 
of the light microscope to focus on different depth of the sample allows for 3D 
imaging of the cell geometry. The famous neuroscientist Cajal (1852-1934) was 
able to identify different types of neurons and also describe connectivity patterns 



and principles of neuronal circuit organization using Golgi's method |5T|. 



Diffusion or transport staining. Diffusion staining techniques enable biologists 
to analyze the projective trajectory of brain regions. For this technique, different 
staining markers are injected into different regions of the brain in vivo. The staining 
is then diffused along the connected neurons. Finally a sample of brain tissue is 
extracted from a different region, in which no marker has been injected. The color 
code in the staining of different neurons in this area then reveals the projection of 
these neurons back to the initial staining areas, providing information about long- 
distance connectivity [34|. The range of possible colors for this method is limited to 
three or four different stainings. 

Multicolor or brainbow. This staining technique does not involve application or 
injection of staining to brain tissue. Instead, transgenic mice are bred to produce 
photophysical fluorescent proteins. A confocal laser-scanning microscope activates 
the fluorescent proteins with a laser beam and records an image with the expressed 
light. Brainbow mice are bred to express three fluorescent proteins of different col- 
ors. By different stochastic expression of these three colors, the single neurons of the 
mice are colored with one out of > 100 labels. The main advantage of this method is 
that it allows to uniquely identify dendrites and axons belonging to the same neuron 
in densely colored tissue [61], see also Figure [4] 

All of these three staining methods allow imaging the geometry of neurons at the 
micrometer scale. The different staining protocols all aim at visually separating sin- 
gle neurons out of the complex and dense neuronal tissue. Visualization techniques 
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for connectomics need to enhance the visual separation further, e.g., by provid- 
ing contrast enhancement and enabling flexible mappings of image data to varying 
amounts of transparency in the transfer function |52| . Especially for the brainbow 
staining it is useful to have visual enhancement of color differences in regions of 
interest where two neurons with a similar staining combination need to be distin- 
guished. For diffusion staining this problem is less pronounced than for brainbow 
data, as typically only three to four easily distinguishable colors are used. But this 
also leads to the challenge of distinguishing two neighboring cells that are stained 
with the same color. This problem also arises in the Golgi stain, as only one color is 
applicable for this staining. Thus visualization needs to focus on providing a good 
impression of the neurons' geometry. The user needs to be able to access the three- 
dimensional structure on different scale levels to infer the connectivity of dendritic 
parts and axons. In order to analyze the neuron geometry further, dendritic and ax- 
onal trees have to be identified and segmented. This task is typically performed 
either semi-automatically or fully automatically with a final proof reading step J55J. 

An additional major challenge for the visualization of microscopy data sets in 
the field of connectomics is the large data volume required to analyze the geometry 
of full neurons. Microscopes typically only record regions of interest at the required 
resolution. Afterwards the acquired images or image stacks need to be stitched into 
one large data volume. While this problem is well known and automatic methods for 
image stitching and alignment exist [25 74], these tools typically work offline, as- 
sembling all images into one large image file for later visualization. But with image 
volumes in the gigapixel range this method is no longer applicable. Instead, visu- 
alization tools are required to perform operations like image stitching, alignment, 
contrast enhancement, and denoising on-demand in the region of interest shown to 
the user. To allow for interactive visualization these operations do not only need 
to be executed fast, but also on multiple scales, allowing the user to zoom in and 
out of the displayed data volume. Recent work by Jeong et al. |43 1 provides this 
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demand-driven visualization approach and combines it with a client server archi- 
tecture. The client can visualize the data with user interaction and annotation while 
computations are performed on a high-performance server transparently to the user. 
Multiple client instances can connect to the same server to allow multiple users to 
access the data at the same time and cooperatively work on the same data set. 



6 Microscale Connectivity 

In contrast to light microscopy, which is limited in its resolution by the wavelength 
of light, electron microscopy enables imaging neuronal tissue at the nanometer 
scale. Hence, electron microscopy is the only imaging modality so far that can re- 
solve single synapses. However, the sample preparation and image acquisition in 
electron microscopy is labor-intensive and time-consuming. As a consequence the 
analysis of the connectivity between single neurons has been limited to sparse anal- 
ysis of statistical properties such as average synapse densities in different brain re- 
gions [2TJ. Little is known about the complete connectivity between single neurons. 
Information about the individual strength of synapses or the number of connections 
between two cells can have important implications for computational neuroanatomy 
and theoretical analysis of neuronal networks |96|. 

Recently, significant progress has been made in the automation of ultra- thin se- 
rial sectioning [37| and automatic image acquisition [22, 53]. These techniques al- 
low neuroanatomists to acquire large datasets of multiple terabytes (TB) in size. 
With a resolution of 5 nm per pixel, and a section thickness of 50 nm, one cubic 
millimeter of brain tissue requires imaging of 20,000 sections with 40 gigapixels 
per image, leading to an image volume of 800 TB. With data sets of this size new 
challenges emerge for automatic computed analysis and visualization techniques. 
Important processing tasks include demand-driven image stitching and alignment, 
cell segmentation and 3D reconstruction, as well as multi-scale visualization and 
multi-user interaction via client server architectures. 

Electron microscopy samples are typically densely stained. While in light mi- 
croscopy sparse staining is necessary to visually separate a cell of interest from 
unstained tissue background (see Section [5]), the fine resolution of electron mi- 
croscopy allows to discriminate structures according to shape, size, and texture. 
Electron microscopy images are limited to gray scale and typically do not have a 
uniform background. Instead, the background is noisy and highly variable, which 
imposes an important challenge for the visualization of electron microscopy im- 
age stacks. The image data cannot be visualized according to gray values alone, 
as the densely stained tissue forms a nearly solid block. Instead, higher order fea- 
tures that discriminate texture and shape, e.g., gradient histograms, are necessary 
to enhance the visibility of different structures of interest in the visualization j42j. 
Ultimately, full segmentation of the image data is necessary to allow the user visual 
inspection of different biological structures, from small structures such as vesicles 
or mitochondria to entire neuronal cells. Figure|5]shows example reconstructions of 
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Fig. 5 Three dimensional reconstructions of neuronal structures from electron microscopy data. 
Left: three dendrites (colored) and all intervening axons (transparent), right: different axons (col- 
ored) with vesicle filled boutons (yellow). 



different neuronal structures from electron microscopy images. A number of soft- 
ware packages have been developed to aid the user in manual segmentation of cell 
structures in the images [15, 30 38). More recent semi-automatic methods greatly 
facilitate this time-intensive process p7]|77]|78][92) . 

Progress has also been made on fully automatic segmentation of EM brain im- 
ages [40 45 48 49 99 101 ]. However, all methods developed so far require manual 
interaction and inspection by users. Thus, visualization tools should not only pro- 
vide the ability to inspect the original EM data and the computed segmentations, 
but also provide a user interface to detect and correct segmentation errors, a process 
called proofreading. 

Another interesting challenge for the visualization of neuronal microscopy im- 
ages is the concurrent display of light and electron microscopy data acquired from 
the same sample. Correlative microscopy is a new developing field, which allows 
inspection of the same neuronal tissue using both light and electron microscopes. 
Thus the fine resolution of the electron microscopy images can be combined with 
the advantage of color staining and information about long-range connectivity in, 
e.g., diffusion stained light microscopy images. Visualization of this data requires 
multi-modal registration of both data sets, which has not yet been addressed for 
correlative microscopy. 

Currently, most research efforts in connectomics at the microscale concentrate 
on the image acquisition and segmentation of electron microscopy images. Little 
research has been done in the visualization of entire connectomes, i.e. the wiring 
diagram of neurons, their types and the connectivity for detailed analysis of neuronal 
circuits. Connectomes, like the manually reconstructed circuit of C. elegans, are 
visualized by connectivity matrices or connection graphs 
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7 Data Integration and Neural Network Modeling 



As described in the previous sections, neurobiological data can be acquired from 
many different sources. Relating these different kinds of data by integrating them in 
a common reference frame offers interesting opportunities to infer new knowledge 
about the relation between structure and function. In this section, we describe two 
approaches and their visualization aspects for such data integration with the purpose 
of inferring functional properties: Brain mapping and network modeling by reverse 
engineering. 



7.1 Brain mapping 

A major goal in neuroscience is to define the cellular architecture of the brain. Map- 
ping the fine anatomy of complex neuronal circuits is an essential first step in inves- 
tigating the neural mechanisms of information processing. The term brain mapping 
describes a set of neuroscience techniques predicated on the mapping of biological 
quantities or properties onto spatial representations of the brain resulting in maps. 
While all of neuroimaging can be considered part of brain mapping, the term more 
specifically refers to the generation of atlases, i.e., databases that combine imag- 
ing data with additional information in order to infer functional information. Such 
an undertaking relies on research and development in image acquisition, represen- 
tation, analysis, visualization, and interaction. Intuitive and efficient visualization 
is important at all intermediate steps in such projects. Proper visualization tools 
are indispensable for quality control (e.g., identification of acquisition artifacts and 
misclassifications), the sharing of generated resources among a network of collabo- 
rators, or the setup and validation of an automated analysis pipeline. Data acquired 
to study brain structure captures information on the brain at different scales (e.g., 
molecular, cellular, circuitry, system, behavior), with different focus (e.g., anatomy, 
metabolism, function), and is multi-modal (text, graphics, 2D and 3D images, audio, 
video) 1 16|54) . The establishment of spatial relationships between initially unrelated 



images and information is a fundamental step towards the exploitation of available 
data 1 7]. These relationships provide the basis for the visual representation of a data 
collection and the generation of further knowledge. 

Databases and atlases. A neuroanatomical atlas serves as reference frame for 
comparing and integrating data from different biological experiments. Maye et 
al. (65 1 give an introduction and survey on the integration and visualization of neu- 
ral structures in brain atlases. Such atlases are an invaluable reference in efforts to 
compile a comprehensive set of anatomical and functional data, and in formulating 
hypotheses on the operation of specific neuronal circuits. 

A classical image-based neuroanatomical atlas of Drosophila is the FlyBrain at- 
la^j spatially relating a collection of 2D drawings, microscopic images, and text. 



1 http : //f lybrain . neurobio .arizona.edu 
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Fig. 6 Visual query for neural projection in the Drosophila brain using the BrainGazer system 1 9 1 . 
Left: The query is specified by sketching a path on top of a Gal4 expression pattern. Right: An 
existing segmented neural projection that matches the query is displayed. 



One approach in generating a digital atlas of this kind is by acquiring confocal mi- 
croscope images of a large number of individual brains. In each specimen, one or 
more distinct neuronal types are highlighted using appropriate molecular genetic 
techniques. Additionally, a general staining is applied to reveal the overall structure 
of the brain, providing a reference for non-rigid registration to a standard template. 
After registration, the specific neuronal types in each specimen are segmented, an- 
notated, and compiled into a database linked to the physical structure of the brain. 
Jenett et al. HI] describe techniques for quantitative assessment, comparison, and 
presentation of 3D confocal microscopy images of Drosophila brains and gene ex- 
pression patterns within these brains. Pereanu and Hartenstein [72] and Rybak et 
al. J80| , described 3D atlases of the developing Drosophila brain and the honeybee 
brain. The Neuroterrain 3D mouse brain atlas [5| consists of segmented 3D struc- 
tures represented as geometry and references a large collection of normalized 3D 
confocal images. 

Visual exploration and analysis. 3D microscopy data is often visualized using 
Maximum Intensity Projection (MIP), which displays the maximum along viewing 
rays. Direct Volume Rendering (DVR) enables better perception of spatial relation- 
ships, but has the disadvantage of added complexity, as an additional transfer func- 
tion is required. It can lead to problems with occlusion particularly when multiple 
channels need to be visualized simultaneously. Maximum Intensity Difference Ac- 
cumulation (MID A) [9] improves this situation by combining the simplicity of MIP 
with additional spatial cues provided by DVR. Wan et al. [ 103] presented a tool for 
the visualization of multi-channel data tailored to the needs of neurobiologists. As 
acquired volumetric data is typically visualized together with segmented structures, 
it is important to avoid occlusions as well as visual clutter. KuB et al. [56 1 proposed 
and evaluated several techniques to make spatial relationships more apparent. 

However, to enable the exploration of large-scale collections of neuroanatomi- 
cal data, massive sets of data must be presented in a way that enables them to be 
browsed, analyzed, queried and compared. An overview of a processing and vi- 
sualization pipeline for large collections of 3D microscopy images is provided in a 
study by de Leeuw et al. [60). NeuARt II 1 13 1 provides a general 2D visual interface 
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to 3D neuroanatomical atlases including interactive visual browsing by stereotactic 
coordinate navigation. Brain Explorer [59 1, an interface to the Allen Brain Atlas, al- 
lows the visualization of mouse brain gene expression data in 3D. The CoCoMac-3D 
Viewer developed by Bezgin et al. [6] implements a visual interface to two databases 
containing morphology and connectivity data of the macaque brain for analysis and 
quantification of connectivity data. An example of an interface to neuroanatomical 
image collections and databases that features basic visual query functionalities is 



the European Computerized Human Brain Database (ECHBD) |32|. It connects a 
conventional database with an infrastructure for direct queries on raster data. Visual 
queries on image contents can be performed by interactive definition of a volume of 
interest in a 3D reference image. Press et al. (75 1 focused on the graphical search 
within neuroanatomical atlases. Their system called XANAT allows study, analysis, 
and storage of neuroanatomical connections. Users perform searches by graphically 
defining a region of interest to display the connectivity information for this region. 
Furthermore, their system supports also textual search using keywords describing 
a particular region. KuB et al. [57 1 proposed ontology-based high-level queries in 
a database of bee brain images based on pre-generated 3D representations of at- 
las information. In the BrainGazer system [9| anatomical structures can be visually 
mined based on their spatial location, neighborhood, and overlap with other struc- 
tures. By delineating staining patterns in a volume rendered image, for example, the 
database can be searched for known anatomical objects in nearby locations (see Fig- 
ure [6]). Lin et al. (62] presented an approach to explore neuronal structures forming 
pathways and circuits using connectivity queries. In order to explore the similarity 
and differences of a large population of anatomical variations, Joshi et al. [44 1 pro- 
posed a similarity-space approach that embeds individual shapes in a meta-space for 
content-driven navigation. 

While these efforts represent promising directions, many challenges remain. As 
noted by Walter et al. |102| , a major goal is the integration of brain mapping data 
with other resources such as molecular sequences, structures, pathways and regula- 
tory networks, tissue physiology and micromorphology. The ever-growing amount 
of data means that distributed solutions are required. The integration of computa- 
tional and human resources gives significant benefits: each involved partner may 
bring computational resources (in terms of hardware and tools), human resources 
(in terms of expertise), and data to analyze. Advances in web technology, such as 
HTML5 and WebGL, for instance, provide new opportunities for visualization re- 
searchers to make their work accessible to the neuroscience community. 



7.2 Neural Network Modeling 

A complete reconstruction of the connectivity at synapse level is currently possible 
for small brain volumes using electron microscopy techniques, but not yet feasible 
for volumes of the size of a cortical column. Oberlaender et al. |67| therefore pur- 
sue a reverse engineering approach: A computational model of a cortical column in 
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the rat somatosensory cortex, consisting of ^16,000 neurons, is created by integra- 
tion of anatomical data acquired by different imaging and reconstruction techniques 
into a common reference system. As the data is acquired from different animals 
in a population, the network represents an "average" cortical column: some model 
parameters are given as probabilistic densities. By generating realizations of these 
stochastic parameters, concrete network models are created. 




Fig. 7 Reverse engineering of a cortical column. Reconstructed dendrites (a) are replicated and 
inserted into the column reference frame according to a given neuron density (b). By determining 
the local structural overlap with axons projecting into the column (c) the number of synapses for 
different post-synaptic cell types can be estimated, (d) Shown are synapse densities for two cell 
types. Figure created from data published in (67). 



The number of neurons and their distribution in a cortical column is obtained 
by automatic counting of neural soma (cell bodies) in confocal images [66|. The 
3D dendritic morphologies of ^100 neurons of different cell types in the column 
as well as axons are reconstructed from transmitted light bright field images J23j. 
The column model is created by generating soma positions satisfying the given neu- 
ron density and replicating and inserting the dendrite morphologies into the refer- 
ence frame according to the given cell type frequency (see Figure|7]i. Differences in 
synaptic densities between cell types can be quantified and visualized [67 1. Based on 
the estimated number of synapses per cell a complete network wiring is established 
to study network function using numerical simulation (58). 

Extracting relevant neurobiological knowledge from such network models is 
a challenging task. Whereas computation of specific quantities for comparison 
with literature results in order to validate the model is straightforward, exploratory 
knowledge discovery within such large, complex networks is not. Easy-to-use tools 
are needed to let the neurobiologist query and visualize the structural and functional 
properties of such networks or ensembles of network realizations. As network mod- 
els are increasing in size, large data handling will be a challenging issue as well. 
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8 Network Analysis and Comparative Visualization 

A recent innovation in neuroimaging is connectivity analysis, in which the anatom- 
ical or functional relation between different (underlying) brain areas is calculated 
from data obtained by various modalities, allowing researchers to study the result- 
ing networks of interrelated brain regions. Of particular interest are comparisons 
of functional brain networks under different experimental conditions and between 
groups of subjects. 



8.1 Network Measures 

For each of the brain connectivity types (anatomical, functional, effective) one 
can extract networks from data obtained by an appropriate brain imaging modal- 



ity 1 11 55 1 . The next step is to characterize such networks. In the last decade a 
multitude of topological network measures have been developed in an attempt to 
characterize and compare brain networks fT2]|46l|79]|9l"| ]. Such measures charac- 
terize aspects of global, regional, and local brain connectivit}^] Examples of global 
measures are characteristic path length, clustering coefficient, modularity, centrality, 
degree distribution, etc. Some of them, such as clustering coefficient or modularity, 
refer to functional segregation in the brain, i.e., the ability for specialized process- 
ing to occur in densely interconnected groups of brain regions. Others characterize 
functional integration, i.e., the ability to rapidly combine specialized information 
from distributed brain regions f79]|9T) . Typical measures in this class are based on 
the concept of paths in the network, e.g., characteristic path length or global ef- 
ficiency (average inverse shortest path length). It is believed that both anatomical 
and functional brain connectivity exhibit small-world properties, i.e., they combine 



functionally segregated modules with a robust number of intermodular links [ 3 ,90 1 . 



The degree distribution can be used as a measure of network resilience, i.e., the 
capacity of the network to withstand network deterioration due to lesions or strokes. 

For characterizing networks on a local scale one uses single node features such 
as in-degree and out-degree, or the local clustering coefficient. Typical regional net- 
work measures are network motifs, which are defined as patters of local connectivity. 
A typical motif in a directed network is a triangle, consisting of feedforward and/or 
feedback loops. Both anatomical and functional motifs are distinguished. The signif- 
icance of a certain motif in a network is determined by its frequency of occurrence, 
and the frequency of occurrence of different motifs around a node is known as the 
motif fingerprint of that node. 



Similar approaches have been used in genomics 



63p5 



and other areas. 
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8.2 Brain Network Comparison and Visualization 

The comparison of different brain networks presents challenging problems. Usually 
the networks differ in number and position of nodes and links, and a direct compar- 
ison is therefore difficult. One possible approach is to compute a network measure 
for each of the networks, and then compare the network measures. However, this 
loses spatial information. For interpretation and diagnosis it may be essential that 
local differences can be visualized in the original network representation |29[87j. 



This asks for the development of mathematical methods, algorithms and visualiza- 
tion tools for the local comparison of complex networks - not necessarily of the 
same size - obtained under different conditions (time, frequency, scale) or pertain- 
ing to different (groups of) subjects. 

Several methods exist for spatial comparison of brain networks, which assume 
that the position and number of network nodes is the same in the networks to be 
compared. For example, Salvador et al. [ 8 1 1 use a brain parcellation based on a prior 
standard anatomical template, dividing each cerebral hemisphere into 45 anatomical 
regions that correspond to the nodes of the brain network. Another possibility is to 
consider each voxel a network node, but in this way the networks become very 
large. Links between the nodes can then be defined by several measures of node- 
node association, such as correlation or mutual information of temporal signals. 
Using the same construction for two or more data sets enables a direct network 
comparison ]106| . 

A method to perform network comparison in the original network representation 
was recently proposed for the case of multichannel EEG by Crippa et al. 1 19 1. This 
approach is based on representation of an EEG coherence network by a so-called 
functional unit (FU), which is defined as a spatially connected clique in the EEG 
graph, i.e., a set of electrodes used in the EEG experiment that are spatially close 
and record pairwise significantly coherent signals p4) . To each electrode a Voronoi 
cell is associated and all cells belonging to an FU are given a corresponding color. 
Lines connect FU centers if the inter-FU coherence exceeds a significance threshold. 
The color of a line depends on the inter-FU coherence. Such a representation of the 
FUs in an EEG recording is called a FU map. FU maps can be constructed for 
different frequency bands or for different subjects (see Figure|8]l. 

Comparison of multiple FU maps can be done visually when displayed next to 
each other, but this method is limited as humans are notoriously weak in spotting 
visual differences in images. An alternative, which is more quantitative although it 
still involves visual assessment to a certain degree, is to compute a mean FU map, 
based upon the concept of graph averaging fl9) . The mean of a set of input FU maps 
is defined in such a way that it not only represents the mean group coherence during 
a certain task or condition but also to some extent displays individual variations 
in brain activity. The definition of a mean FU map relies on a graph dissimilarity 
measure that takes into account both node positions and node or edge attributes. 
A visualization of the mean FU map is used with a visual representation of the 
frequency of occurrence of nodes and edges in the input FUs. This makes it possible 
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Fig. 8 FU maps for multichannel EEG coherence visualization. Brain responses were collected 
from three subjects using an EEG cap with 1 19 scalp electrodes. During a so-called P300 experi- 
ment, each participant was instructed to count target tones of 2000Hz (probability 0.15), alternated 
with standard tones of 1000 Hz (probability 0.85) which were to be ignored. After the experiment, 
the participant had to report the number of perceived target tones. Shown are FU maps for target 
stimuli data, with FUs larger than 5 cells, for the l-3Hz EEG frequency band (top row) and for 
13-20Hz (bottom row), for three datasets. 



to investigate which brain regions are more commonly involved in a certain task, by 
analyzing the occurrence of a FU of the mean graph in the input FUs. 

In p9) the graph averaging method was applied to the analysis of EEG coher- 
ence networks in two case studies, one on mental fatigue and one on patients with 
corticobasal ganglionic degeneration. An extension of the method to resting state 
fMRJ data was presented in |20) . 



9 Conclusions 

There is currently great scientific interest in connectomics, as it is believed to be an 
important prerequisite for understanding brain function. As much of the data for ob- 
taining neural connectivity is image-based, visualization techniques are indispens- 
able. Great effort has been put recently into extraction of connectivity information 
from images, integration of multimodal information into reference systems, and vi- 
sual analysis of such data and systems at different scales. These efforts will need to 
be intensified in the future, as data is being produced at a much larger scale, also 
by new image modalities. New methods to integrate this data across modalities and 
scales to attain the ultimate goal, a description of the human connectome, will be 
the main challenges for visualization in connectomics. 
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